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Introduction 


Universidade Aberta (UAb), being a distance teaching university, is particularly suited to tackle some of the 
recommendations concerning the assurance of accessibility to education expressed in the "Education and 
Training 2010" goals (European Commission, 2008a). Meanwhile, the University's strategic plan for 
2006/2010 decided to implement a fully virtual innovative teaching and learning methodology. The 
pedagogical model (Pereira, Quintas-Mendes, Morgado, Amante & Bidarra, 2007) adopted by the 
University is "strongly dominated by the valuing of students' social and communal integration, 
personalized monitoring of their learning and the respect for the particular life experience of each student" 
(p.8). This pedagogical model, in use since 2007, is based on four cornerstones: 1) student-centered 
learning, 2) flexibility, 3) interaction and 4) digital inclusion, aligned with the key competences required for 
learners in this knowledge-based society (Directorate for Education, Employment, Labour and Social 
Affairs, 2002). According to the University pedagogical model (Pereira, 2007) "these principles guide the 
organization of instruction, planning, the design and management of activities for the students, the type of 
materials to develop and the nature of the assessment of competences" (p. 11,12). 

1. Student centered learning - at the core of the model is "the student as an active individual, builder 
of his own knowledge, engaged in and committed to his learning process within a learning 
community" (p. 12). In this way, students are enabled "to develop their autonomy and creativity, to 
acquire the capacity to monitor their outcomes and to plan their education, preparing them for 
lifelong learning" (p.12) 

2. Flexibility - this cornerstone takes into account the fact that most UAb students are "adults with 
professional, familiar and civic responsibilities, active citizens in society" (p.13). For that reason, 
"the emphasis on asynchronous technologies makes it easier for students to manage online access 
time, individual research, study and exploration of subject and/or interaction with the teacher and 
the other students" (pp. 13,14) 

3. Interaction - going beyond traditional student-contents and student-teacher interaction, the 
University pedagogical model also strongly supports "student-student interaction through the 
creation of discussion groups within each virtual class" (p. 14). 

4. Digital inclusion - this cornerstone is particularly important in the context of a fully virtual learning 
environment, such as the one offered. In this way, the University is promoting the development of 
competences in the use of information and communication technologies, especially in the case of 
adults who, even without having such ICT competences, wish to enrol in a higher education 
institution. In this context, being a student at Universidade Aberta is a factor for social inclusion, 
contributing to digital literacy (p. 14,15). To facilitate this process, all UAb programs include a free 
introductory module, where new students acquire specific e-competences (Schneckenberg, 2008) 
necessary for online learning before starting their studies. 

The dramatic changes that are taking place at the University call for close monitoring of its implementation 
and for applied research on online methodologies, which call for new open and distant learning assessment 
strategies. For this reason, a new research unit was created at the University: the Distance Education 
Laboratory (LEaD - Laboratorio de Educagao a Distancia). Some of the LEaD goals include: applied 
research focused on virtual pedagogies; the development and testing of pedagogical and technological 
innovative solutions; gathering, sharings and disseminating knowledge in the area of e-learning; 
promotion of advanced professional development for course designers and e-tutors. 

The "E-leaming and Success" project is a 2 year (2007/2009) LEaD multidisciplinary project with four 
research teams focused on: development and testing of e-resources (group A); continuous assessment and 
competence development (group B); resources for foreign language teaching and assessment (group C); 
student success and satisfaction (group D). 

Theoretical Framework 


The implementation of the new pedagogical model forced, and at the same time called for, the use of new 
assessment strategies aligned with the most recent paradigm for assessment design in online learning 
(Birenbaum, 2003; Brown, Bull & Pendlebury, 1997; Mateo & Sangra, 2007). To implement this change it 
is necessary that all teachers become knowledgeable in these new alternative approaches to assessment in 
higher education (Maclellan, 2004). 
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The current research, being part of a larger project, is focused on assessment issues concerning 
undergraduate education. The discussed pedagogical model calls for two main types of instruments to be 
used in undergraduate fully online courses: e-folios and p-folios. The e-folio "is a short digital document 
elaborated by the student and published online to be visualized by the teacher, and should clearly 
demonstrate that the student acquired or developed a given competence" (Pereira, etal. 2007, p. 19). Here 
are some examples of what an e-folio may include: a critical reflection about the student's own learning; a 
report about field work; a problem solving activity; a reading review; or the production of an artifact. Being 
introduced as an instrument for continuous assessment of the competences promoted in a particular 
curricular unit, students are expected to produce two or three e-folios per semester. The e-folios may be 
complemented by a p-folio that takes place in a face-to-face setting. The p-folio may take the form of "a set 
of questions defined by the teacher, or other forms, such as the presentation of a project or a report, 
according to the competences to be developed by the students" (Pereira et al. 2007, p. 19). 

The presented assessment model consists of a combination of different assessment methods, including 
alternative forms of assessment (the e-folio). In this way, we are promoting the development of a 
Competence Assessment Program (CAP) (Baartman, Bastiaens, Kirscgner and van der Vleuten, 2007). 
According to these authors, "CAPs combine elements of the testing culture and the assessment culture. 
Newer forms of assessment are not regarded as alternative to traditional tests, but as complementary to 
them" (Baartman et al., 2007, p. 117). Using just one assessment instrument is insufficient to make a clear 
judgement about one's competence. It is necessary to conceptualize the assessment of competences by 
taking into account that they represent a complex concept that includes knowledge, skills, aptitudes and 
values. Thus, there is a clear need to develop a CAP including various assessment methods. 

In order to implement this project it was fundamental to start by reflecting on the concept of competence to 
be adopted. The agreed, ongoing, construct for competence in use was based on the work of the DeSeCo 
project (Directorate for Education, Employment, Labour and Social Affairs, 2002) that presents 
competence as the capacity to successfully respond to individual, and societal, solicitations or to 
successfully perform a task or activity, requiring the mobilization of knowledge (both implicitly and 
explicitly), abilities, attitudes, emotions and values. Competence is defined as the capacity to respond to 
individual, or societal, demands in order to perform an activity or complete a given task (Lizzio & Wilson, 
2004; Messick, 1984; Perrenoud, 1997, Tillema etal, 2000). Competences are observed as a result of 
individual's actions in a given context. They are developed through acting and interacting in both formal 
and informal educational or professional contexts, and require going beyond the mere reproduction of 
acquired knowledge. At its highest level, this conceptualization of competence implies to choose and adapt 
from within the acquired processes those ones necessary to solve an unknown complex task or problem 
(Rey, Carette, DeFrance and Kahn, 2005). Competences represent a complex web of knowledge, capacities 
and attitudes that need to be used in order to solve a problem (Perrenoud, 1997). Moreover, as LeBoterf 
(2005) points out, competences and competitiveness are two words with a related etymology: early in 1999, 
the Bureau International du Travail stated that the level of competence is one of the main advantages 
needed in a globalized market. 

The assessment of competences also requires a renewed approach where knowledge, abilities and attitudes 
are integrated (Baartman et al., 2007). Necessarily, it should make use of a variety of different assessment 
strategies and tools (e.g., Chester, 2003; Dierick & Dochy, 2001; McConnell, 2004; Maclellan, 2004), so as 
to better assess performance in authentic activities that should be as similar as possible to the contexts in 
which the competences will be implemented. Furthermore, a strong emphasis to the diagnostic and 
formative characteristics of assessment should be given, so as to better promote the participants reflection 
and competence development. 

In order to characterize the types of competences being developed and assessed at UAb within the scope of 
the newly adopted pedagogical model, group B created a framework comprising four main types of 
metacompetences: problem solving; group work; metacognitive; and fluency in ICT use (Table 1), 
supported by the adopted pedagogical model, the literature and the researchers as a reflection of their 
professional practice. 


Table 1. Types of Competence 


Metacompetence 

Competence description 

Problem solving 

1. 

Observe/gather/select / identify / analyze 
phenomena/situations/cases/documents according to a 
theory/ approach/perspective 


2. 

Compare/justify/confront/question perspectives/approaches/viewpoints 


3- 

Decide/state/make judgements 


4- 

Plan/organize/design an intervention/product/service 


5- 

Argument / question/ discuss/interpret 


6. 

Present/synthesize/summarize/describe/characterize in diverse 
contexts about situations/phenomena/cases 


7- 

Plan/present a discourse (audio, video, text) 

Group Work 

1. 

Share information and resources (documents, summaries, e-folios) 


2. 

Exhibit solidarity/peer help 


3- 

Promote group action/interaction (to make suggestions/proposals) 

Metacognitive 

1. 

Reflect upon one's trajectory/ learning process/knowledge/goals 


2. 

Analyze/identify gaps of knowledge/wrong turns/shortcomings 


3- 

Self-questioning, explicit/describe learning obstacles 

Fluency in ICT use 

1. 

Efficient use of ICT tools 


2. 

Plan/create products to be available online 
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Aims and Questions 

The current project intends to research and develop the pedagogical model for undergraduate courses at 
Universidade Aberta, and to identify strategies that promote the students' learning and success. In 
particular, we are concerned with the development of instruments and approaches to continuous 
assessment, targeted at the promotion of specific competences. In this context, two main research 
questions were asked: 1) What relationship is there between the concept of competence being used in the 
e-folios and our current working definition? 2) What are the main characteristics of the e-folios? 

Methodology 

The methodology used in the whole project is design-based research. Bereiter (2002) defines design-based 
research as any kind of research that produces findings that are fed back into further cycles of innovative 
design. The use of design-based research includes five main principles: (i) strategic investigation as an 
answer for identified problems; (ii) contextualized research focused on the identified problems; (iii) 
research targeted at innovation and practical applications; (iv) a practice conceived and oriented by 
research goals; and (v) the existence of research and assessment cycles for the planned interventions. 
Anderson (2005) specifically refers to the suitability of applying design-based research methods to 
educational innovation contexts, in general, and to distance education, in particular. 

Next, we present (Figure 1) the Design Based Research (DBR) methodology used by the project's group B: 
continuous assessment and competences. Initially, there was a discussion among all research project team 
members, based on the literature review about the concept of competence and aiming its clarification in the 
university's online pedagogical model context. 


Revtewof Competence- Clarification of concepts 

related literature of competence 

Conceptual Ffameworir draft 


WorUng definition for the Data Sample 

concept of competence t**pmm** Collection 


Development of the 
an at/si $ pic 


feanfcjancn, Kvtskxi and e-fal« analpfe based on the 

exwnstonoftbeused de«doped f«™wori< 

framewwfc 


Aral framevMorit 
Conclusion of the D8R 1st cycle 
launditog of the 2nd DBR cyde 


Figure 1. Design Based Research Methodological Steps taken by "e-Learning and Success" Group B 

After the group discussion and clarification around the concept of competence and the creation of the 
framework (Table 1), the second stage was focused on the analysis of a pool of 35 e-folios collected within a 
sample that several teachers voluntarily made available, representing seven different scientific areas: 
education (n=io), languages (n=io), culture (n=4), literature (n=3), natural sciences (n=2), economy 
(n=2), and documental sciences (n=4). 

Simultaneously, the research team created an analysis grid centered on the following criteria: the course 
competences; the e-folio competences; types of competences; the assignment; types of tasks; assessment 
criteria; instructions; resources; timetable; and structure. This grid of analysis resulted from a previous 
exploration of a subset of e-folios, under the lenses of the theoretical framework and its application to 
didactical contexts, as well as from the presented research questions. 

The ongoing process of re-evaluation of the framework, of the e-folios analysis and of the development of 
the analysis grid led to the final framework, to the conclusion of the DBR 1 st cycle and to the launching of 
the next DBR cycle. 

In order to triangulate data, the team was organized into smaller 3/4-person groups: each group analyzed a 
set of e-folios which was afterwards analyzed by another group. Finally, the results were discussed among 
the members of the team. During this phase, the group felt a strong need to recalibrate, discuss and 
upgrade the criteria being used to fill out the analysis grid and continuously readjust inter-codification. 

Results and Analysis 

The collected sample was constituted by 35 e-folios, as mentioned above. From this initial sample, 4 
e-folios were excluded — 2 because they required the students' participation in an online discussion forum 
which was then assessed but is not in accordance with the adopted pedagogical model (Pereira et al., 
2007), and 2 others because they were not considered to be assessing competences as they competences 
were defined by the group; instead, they merely required the students to follow a list of procedures. 

The first major result is the fact that the group's definition of competence was always able to accommodate 
to the types of competences being assessed in the analyzed e-folios. However, our working definition of 
competence also proved to be considerably larger and to encompass a wider range of metacompetences 
than what is actually being assessed. 

To answer our first research question concerning the relationship between the concept of competence 
being used in the e-folios and our current working definition, from the 31 e-folios included in the final 
sample all of them presented a concept of competence that was framed by our larger framework. From the 
assessed competences, 136 were included in the Problem Solving metacompetence, o in the Group Work, 3 
in the Metacognitive and 3 in the Fluency in ICT use. 
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The absence of competences related to Group Work, in this analysis, is a direct consequence of the 
individual nature of the e-folio task. However, this metacompetence is promoted in other moments of the 
course, when the students are invited to participate and collaborate through the online forums, among 
themselves and with the teacher/tutor. 

The reduced number of competences related to the Fluency in ICT use may be a consequence of the 
existence of a very large number of indirect requests related to ICT in this learning environment, 
consequently its use was not explicitly made or requested; in fact, only 3 e-folios contained them. 
Moreover, most of the sampled undergraduate degrees include Curricular Units specifically in the area of 
ICT, in accordance with the requirements of the Bologna Process (European Commission, 2008b), that 
have not been included in this sample and that are particularly targeted at the development of ICT 
competences. 

Related to the Metacognitive metacompetence there were also only 3 e-folios assessing competences 
associated with the analysis of knowledge gaps. This may be due either to the fact that problem solving is 
more valued at the beginning of an undergraduate degree, or to the perceived difficulty felt by the teachers 
to assess metacognitive competences. 

The large majority of the competences being assessed in the e-folios are framed by the metacompetence of 
Problem Solving. In particular, competence 1 (Table 2) was object of assessment in most of the analyzed 
e-folios (77%), appearing even more than once in several occasions. Competence 6 (Table 2) was the 
second most frequent competence, being assessed in a large number (45%) of the analyzed e-folios. 
Competences 3 and 7 were also assessed a reasonable number of times. Competences 2, 4, and 5 were all 
also assessed but less frequently. 


Table 2. Frequency of observed competences 


Competence 

Frequency 

1. Observe / gather / select / identify / analyze 

50 

phenomena/situations/cases/documents according to a 
theory/approach/perspective 

7 

2. Compare/justify/confront/question perspectives/approaches/viewpoints 

16 

3. Decide/state/ make judgements 

4 

4. Plan/organize/design an intervention/product/service 

10 

5. Argument/question/discuss 

30 

6. Present/synthesize/summarize in diverse contexts about 

19 

sit uations/phenomena/ cases 


7. Plan/present a discourse (audio, video, text) 



Finally, it was observed that there was a certain diversity concerning the types of competences being 
assessed in each e-folio: there were 5 e-folios that only assessed one type of competences, 10 e-folios that 
assessed two types, 11 e-folios that assessed three types and 5 e-folios that assessed four types. It is 
important to analyze up to which point this variability may be related to the nature of the task being asked, 
the scientific area, and/or the competences defined for each particular curricular unit. 

To answer our second research question we attempted to create a descriptive characterization of the 
e-folios. Table 3 presents a summary of the main characteristics common to the sampled e-folios. As it 
might be expected, most of the e-folios included instructions related to the task at hand, the assessment 
criteria, and the needed resources. 


Table 3. Characteristics of the e-folios 


Characteristic 

Frequency 

% 

Instructions detailing the steps to take to complete the task 

28 

90 

Instructions related to formal aspects 

23 

74 

Assessment criteria 

26 

84 

Resources 

25 

81 

Timeframe 

17 

55 


The types of tasks also constituted another source of variation. Table 4 presents a summary of tasks in the 
31 analyzed e-folios. It is important to notice that some of the e-folios required more than one task (for 
example, an essay and a practical exercise). From this analysis, the pre-eminence of essay and practical 
exercises tasks become clear. This may be due to the fact that the gathered sample is mainly represented by 
e-folios from the social sciences and the language/literature/culture learning areas. 

Table 4. Type of task of the e-folios 


Task 

Frequency 

% 

Essay (writing composition, reflection, analysis) 

20 

65 

Practical exercise (application, demonstration) 

10 

32 

Construction of an artifact (blog, pictures, poster, video, slideshow) 

5 

16 

Critical comment (stating an opinion) 

3 

10 

Project (designing a planned action) 

1 

3 
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Report (of fieldwork, of experimental work) 

1 

3 

Reading review (reading guide summarizing the main ideas in a text/excerpt) 

1 

3 


In 25 (81%) of the analyzed e-folios the students were required to make use of some kind of resource that 
was explicitly mentioned. Such set of resources are: texts included in the e-folio (6) (to contextualize or to 
be used as a tool for reading comprehension); other texts (12) (newspaper articles, research journal articles 
or book chapters/excerpts); links to external sources (10) (such as blogs, podcasts or videos); and a TV 
program. This description illustrates the teachers' preference for traditional types of written sources. 

A large majority of the e-folios (26 representing 84% of the total sample) also made explicit several aspects 
concerning the way that the assessment was to be materialized, while 16% did not include any references to 
the assessment process. However, the assessment criteria were not always made explicit: in some cases 
only a scoring rubric was presented, while in others some of the content requirements were mentioned. 

The level of specification varied considerably. 

To conclude, one final aspect of the e-folios that was considered for the analysis was the overall structure in 
which it it is organized. In particular, there was an attempt to clarify the constituting sections of each 
e-folio. The number of sections ranged from one to nine, in the cases with a higher number of 
specifications. Examples of the sections include: description of the activity; timetable; resources; 
objectives; competences; calendar; formal instructions; assessment criteria; estimation of student time 
required to complete the e-folio. Others presented a simpler structure, closer to a traditional written 
examination/test, and included only 2 to 4 sections (such as reading comprehension and written 
comprehension). In our sample, two subsets emerged: 61% of the e-folios contain four or less sections, 
while 39% of the e-folios are constituted by seven to nine sections. This appears to indicate two different 
conceptions of the way to present the e-folio to the students, one with a very structured and specified 
organization, and another exhibiting the opposite tendency, i.e. not valuing a detailed structure. As a 
matter of fact, the adopted pedagogical model (Pereira et al., 2007) is sufficiently open to allow a wide range 
of tasks and structures. 

Conclusions and Implications 

From the analyzed e-folios, it becomes clear that the assessed competences were all framed by the concept 
of competence assumed by this research group, although they are not always presented with a clear 
formulation. The most valued type of competence was clearly "problem solving". Within this 
metacompetence, most of the e-folios required the students to "gather and analyze information". The 
majority of them required the "synthesis of perspectives", while others asked the students to "plan and 
design a discourse", and demanded the students to "make judgements". Finally, few e-folios required 
students to "argument and discuss situations/cases", to "confront viewpoints", or to "design a project". 

Although the assessment of the fluency in ICT use competence only appeared in a reduced number of 
e-folios, it is an underlined assumption to the concept of the e-folios, given their digital nature. 
Furthermore, several of the analyzed e-folios explicitly required online research tasks, while others 
required the use of diverse software, which is aligned with the "efficient use of ICT tools" competence 
(number 14 in Table 1). The absence of the metacompetence "group work" in the analyzed e-folios can be 
considered an expected result, given the individual nature of the work required in them. 

In this sample of e-folios only a small number of situations promoting the assessment of metacognitive 
competences were observed. On the one hand, this may be due to the small sample and because the 
analyzed sample was drawn from curricular units belonging, in general, to the 1 st year of several 
undergraduate degrees; on the other hand, the paradigm shift towards a student-centered learning 
environment is also recent at the University and probably responsible for the lack of investment in the 
development and assessment of metacognitive competences. Bearing in mind the fact that this is an 
exploratory study, we stress the relevance and need of increasing the sample size, particularly the number 
of e-folios from 2 nd and 3 rd year curricular units, as well as e-folios from other scientific areas. 

The implemented pedagogical model (Pereira et al., 2007) has improved and expanded the possibility of 
using alternative assessment strategies. These new alternative assessment strategies can be implemented 
not only in the e-folios, but also in the types of resources and the tasks designed by the teachers. However, 
the e-folio is still a very recent tool, with only one year of implementation, and this fact may be responsible 
for the strong emphasis on textual based resources and essays. The use of textual based resources is often 
appropriate and recommended to develop problem solving types of competences. Nevertheless, it is 
expectable that in the future, with the development and consolidation of the change process, the variety of 
resources and types of tasks being used will increase considerably. 

There is also a clear awareness from the teachers of the need to include information about their assessment 
practices in the e-folios. However, a lack of clarity about what to include about assessment seems to persist. 
This aspect is particularly relevant given its pertinence to student learning (Birenbaum, 2003; Brown, Bull 
&Pendlebuiy, 1997). 

Moreover, there is not a clear structure pattern for the analyzed e-folios, disclosing several interpretations 
of what an e-folio may be. However, in particular areas such as foreign languages learning, the emergence 
of a pattern of e-folios with similar structures is apparent. Furthermore, based on the produced analysis, 
the authors believe that particular sections in the structure of the e-folios are especially important and 
ought to be explicit for the student performance and learning. These sections may include: the 
competences being assessed; the assessment and scoring criteria; the task that the students are expected to 
perform; and formal requirements to be met. This is mainly important in a distance learning framework, 
where the student is not physically present, and it becomes necessary to provide him with clear prompts 
about how to organize his/her work and so promote a student-centered learning environment (McConnell, 
2004). 

The implementation of the Bologna Process (European Commission, 2008b) has presented to higher 
education the difficult challenge of promoting and assessing the development of competences. This holds 
true for HE institutions in general, and for Distance Education in particular. With this project and 
especially with the e-folio tool we hope to contribute to bridge this gap with a tool that gives emphasis to 
the type of competences that are being developed and to the strategies that can be used to assess them. 

Finally, the sharing of these preliminary results may assume a relevant role for the teachers who are 
currently using this new tool, or will be in the future, in order to promote a reflective analysis about their 
own assessment practices. 
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